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ABSTRACT 



Although the importance of systematic evaluation of 



educational policy and practice is well established, various problems 
confo. i the intention of evaluating .regional or national activity 
with regard to computer-related activities in education. At least two 
of these problems relate to a general conception of the appropriate 
entry points for evaluative activity, while at least two ether 
problems relate specifically to computer-related policy and practice. 
At least one additional layer of difficulty applies when the 
ts/aluation of such activity involves an external evaluator working 
internationally, particularly in the less developed cc ntries. Each 
of these classes of impediments is briefly analyzed and a general 
model for evaluating regional and national-level computer-related 
educational activity is discussed with respect to the classes of 
impediments and illustrated in the context of an ongoing external 
evaluation of a national educational software development project in 
the Netherlands. This activity — titled the "POCO Project" after its 
Dutch name nieaning software development for computers in 
education — has the organizational goal of developing a first set of 
18 software packages in 18 months. It focuses on developing 
educational software that can be used by teachers in a meaningful way 
during their regular teaching activities and v;ith such frequency as 
to strengthen the teachers* perceptions that using such packages is 
an effective and efficient response to an educational need. The 
particular advantages of an outside evaluator are emphasized in this 
description of the POCO Project. (Author/SLD) 
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ABSTRACT 



Although the importance of systematic evaluation of educational policy and 
practice is well established, various problemi> confound the intention of 
evaluating regional or national activity with regard to :omput€ r-related 
activities in education. At least two of these problems relate to a general 
conception of appropriate ent?7y points for evaluative activity while at 
least two other problems relate specifically to computer-related policy and 
practice. At least one additional layer of difficulty applies when the 
evaluation of such activity involves an external evaluator working 
internationally/ particularly in less developed countries. Each of these 
classes of impediments is briefly analyzed; also, a general model for 
evaluating regional and national-level computer-related educational 
activity is discussed with respect to the classes of impediments and 
illustrated in the context of an ongoing external evaluation of a national 
educational software development project in The Netherlands. 
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Problems and Perspectives on the Evaluation of 
Regional and Na tional Computer-Related Educational Activity 



Problems Associated with the Evaluation of Regional or 
National Computer-Related Educational Activity 

Very often, decision makers, involved with computer-related po'iicy or 
practice, especially at the regional or national level, express the 
conviction that it is "too early" to evaluate these sorts of activities. 
This perspective suggests the decision makers have an unproductively narrow 
view of evaluative activity, focussing on it as a summative, endpoint 
activity. Such a conception overlooks the considerable value evaluation can 
bring to the formative stages of activity. The evaluation models developed 
by Stake (see, for example, 1973) clearly support the place of logical, 
analytic assessment of the assumptions and intentions of any educational 
activity, and yet this aspect of evaluation appears to be undervalued in 
the field. Accompanying the perspective that it is too early to evaluate 
computer-related educational activity is the view that an evaluation will 
somehow deflate the momentum accompanying exploratory activity in this 
ere;*.. Again, this type of opinion reflects a narrow view of evaluation, in 
this case as a perjorative, judgemental exercise rather than as a 
constructive, collaborative procedure. 

Another general problem impeding the assignment of evaluative activity 
to computer-related policy and practice is the perceived lack of a 
practical and cost-effective methodology for conducting such an evaluation. 
Evaluation activity is often seen as expensive and cumbersome, and 




frequently is of little applicability when long time delays ^.eparate it 
from its usefulness in the field. 

This last point is particularly critical with respect to computer- 
related activity, Such activity is changing and evolving so rapidly that 
indeed it can be seen as a "moving tc.rget" with respect to evaluation. 
Unless evaluation can be brought to bear quJ.ckly in a situation, its 
prescriptions are likely to be outdated before they are distributed. 

At least one other characteristic of computer- related activity 
distinguishes it from most other evaluation focuses: computer-related 
activity is frequently "belief driven" rather a goal driven (Bakei, 
1988). Baker notes that general policy for computer use in schools is often 
unrelated to specific student achievement goals, but instead is "motivated 
by a more general desire to improve educational quality, impelled by a set 
of beliefs that the technology will somehow affect the quality of the 
student's learning" (p. 2). In developing countries, these beliefs may be 
driven more by the desire to not fall further behind, relative to more 
developed countries, than by a proactive plan for particular computer- 
related strategies. 

with these, and other conceptual and practical difficulties, it is not 
surprising that evaluation activity is far less prevalent than 
implementation activity with respect to computer-related technology in 
schools. What could be h-^lpful is more awareness of an evaluation strategy 
that is simple and flexible enough to be used at any entry point with 
respect to computer-related policy and practice but still powerful enough 
to supply timely and relevant feedback to central decision makers. 
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A General Model for the Evaluation of 
Computer-Related Educational Activities 

A very simple model, based on Stake's 1973 concepts of congruency and 
contingency asi?essment, has served as the framework for a series of 
evaluation projects involving regio^wide or nationwide computer activity. 
The model is shown in Figure 1. The major features of the model relate to 
the possibility for evaluation of the logical relationships among 
assumptions about the educational situation, intended goals, and intended 
means (represented by the arrows between Boxes A and 3, A and C, and B and 
C) ; the congruence between observed and intended procedures (representee by 
the arrows between Boxes C and D) ; and the interpretation of outcomes (as 
represented by the four arrows radiating from Box E) , This distinction 
allows the discrimination of 'theory breakdown' and 'performance breakdown' 
(Suchman, 1976) . The constructive use of the model occurs through its 
iterative application; intentions of the system — with regard to basic 
assumptions, or goals, or operational strategies ~ can be reexamined and 
altered at any time, triggering a new cycle of assessment of logical 
relationships and/or congruences between intended and actualized activity. 
In this process, assumptions can be clarified and goals become better 
articulated, an outcome that Baker sees as a major contribution of 
evaluation of belief-driven computer-related activities. 
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Intentions Observations 



B 



L; 



Assumptions 



I 



ir- 



Objectives 
or 

Expectations 



I 
I 
I 



Are 

e>;pectations 
being met? 



/ I If not, why? 



What is 
is planned 
to happen? 



D 



What actually 
happens? 



liSlL ^^ 1- A simple general model for educational evaluation, adapted f 
Stake (1973) . 
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The Model Applie d to a National Computer-Related Educational Project 

The model has been used as the bc.sis for an ongoing, external evaluation of 
a national computer-related project in The Netherlands. This activity, 
called the "POCO Project" from its Dutch name "Programmatuur Ontwikkeling 
voor Computers in het Onderwijs" (software development for computers in 
education) was established by the Ministry of Education in The Netherlands 



in June 1987. The project has as its goal the development of educational 
software 

that can be used by teachers in a meaningful way during 
their regular teaching activities, and that can be used 
with such frequency as to strengthen the teachers' 
perceptions that using such packages is an effective 
and efficient response to an educational need. 
(Policy Note/ Minister of Education, June 11/ 1987, p. 2) . 

The project was also given the organizational goal of completing a first 
set of 18 software packages in 18 months ("cycle 1") and a subsequent set 
of approximately 25 more packages during the following two years (cycle 2) . 
For political reasons/ the Minister also established that the Project must 
involve collaboration with the established educational publishers in The 
Netherlands (to minimize their fears that a subsidized government project 
would complete with their own commercial initiatives) . 

A final goal for the project emerged during the initial elaboration of 
the Minister's directive: 

The POCO Project has to promote products that are marketable/ 
both to the internal Dutch market but also in foreign 
countries. Through the incomes of these "salable" products 
(both the software and the approach used for developing the 
software) the project can become a self-supporting 
activity (Moonen/ 1987/ p. 3) . 
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A critical point for the project was to be the transition between Cycle 
1 and 2. At that point/ a decision about continuation of the project under 
its current management would be made by the Ministry. With approximately 
S 12 million U.S. committed to the overall project/ the seriousness of 
preparation for the post-Cycle 1 evaluation can be appreciated. 
Conseq*jently^ the director of the group given the contract for the project 
felt it desirable to involve an external evaluator throughout the Cycle 1 
period. The combination of external evaluation and ongoing evaluation 
reflected two sets of needs: to provide an informed critical commentary on 
project activities from the perspective of someone from both outside the 
project and outside the Dutch educational community/ and to interprete 
those activities relative to the short-term and long-term goals of the 
project. 

A plan for the evaluation was developed by an external evaluator who 
would make periodic visits to the project over its first 18 monthS/ and 
present/ at specified intervals/ eight evaluation reports (Collis & Berger/ 
1987) . An interesting component of the evaluation fror» a practical 
dimension was that the evaluator maintained a regular data collection 
stream through electronic "interviewing" via telecommunications during the 
times she was not visiting the project. 

An immediate challenge of the evaluation was how to access progress 
toward goals that could only be directly assessed long after the project 
was completed — goals about developing software that will be "in frequent 
use" by teachers and that will be profitable in the educational market 
outside The Netherlands. The development of feedback loops (or 
"confrontation" opportunities) between representatives of those eventual 
target groups — teachers from The Netherlands and educational decision 
makers from outside The Netherlands — was seen as a means by which the 
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project could anticipate the long-range response from these groups (Moonen, 
198 #) . Having frequent "confrontations" would also allow adjustments to be 
made to the packages during the design and development phase in order to 
fvrther enhance their eventual irarket appeal. In consequence a major focus 
of the ongoing external evaluation of the project was the degree to which 
these sorts of confrontations took place (Collis & Bergers, 1987) . 

A second major focus of the evaluation involved the interpretation of 
project slowdowns. In a project of the scope and complexity of POCO, 
involving/ in various wayS/ hundreds of people in diverse groups 
(educational publishers, teachers, curriculum specialists/ educational 
technologists/ commercial software houses) working on, in effect/ 18 
different projects/ it is ineviv.able that project execution would not 
always proceed as planned. However/ it was perceived to be highly valuable 
both in an inonediate sense in Cycle 1 and for planning relative to Cycle 2, 
that a distinction be made between slowdowns rooted in "theory" or 
"logical" problems and slowdowns rooted in implementation specifics. When 
execution decisions form the major basis of project slowdowns/ the 
evaluator could offer comments on alternative execution strategies. 
Relative to theoiry or logical slowdowns/ however, the evaluator could also 
help the management team identify elements of their basic planning which 
might be modified in Cycle 2. AlsO/ the evaluator could elaborate more 
globally on basic implications of the original assumptions and goals of the 
project. 

For example/ a fundamental tension exists between the goal of actively 
involving educational publishers in the project and the goal of creating 
products of interest to buyers outside The Netherlands. Involving 
publishers meant creating products th^\t reflected existing Dutch texts and 
methodology/ thus limiting the portability and innovativeness of the 
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software. It would be inappropriate to eventually indict the POCO 
management for not meeting the dual goals of publisher accorrtmodation and 
external saleability when this result is more a logical consequence of 
initial assumptions about the project then is is something that could be 
related to proje^.'. management execution decisions. Thus, the continual 
application of the model shown in Figure 1 allowed the evaluator to 
categorize interpretations of ongoing projecc activity and to offer 
constructive suggestions based on those interpretations (Collis & Moonen/ 
1988) . 

Because the POCO evaluation was ongoing its interim reports could focus 
on this distinction between "theory" and "execution" relative to specific 
aspects of the project at the times they were most pertinent to the 
project. Some of the evaluation focuses included: 



- Organizatica of appropriate staff 

- Development of a priority list of software topics 

- Development of product descriptions 

- Lines of communication and responsibility within the project 

- Interactions with publishers 

- Technical development of the software 

- Development of print materials to support the software 

- Field testing of products 

- Ongoing field involvement ("confrontations") 

- Ongoing information dissemination about the project/ 
nationally and internationally 



The evaluation had at least one other contribution. The periodic 
reappearance of the evaluator with the mandate to ask individuals about 
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their intentions and activities served as a catalyst to better project 
self -awareness. This is a particular contribution of an external evaluator 
— to be sufficiently outside the system that it does not seem unnatural at 
various points in time to ask specific questions about execution decisions 
and intentions in the way it would if the evaluator were part of project 
staff. The stimulus of the evalutor's visits ensured the project had 
regular periods of self-inventorization, something it would probably not 
have taken time to do, at least on such a regular basis, without such a 
stimulus . 

Although Cycle 1 has been given an extension beyond its original 18- 
month timeline (slowdowns were much more extensive than anticipated) , the 
ministerial evaluation relative to Cycle 2 has been recently completed and 
the POCO management have been given approval to proceed with Cycle 2. The 
expensive documentation provided by the external evaluator in the eight 
Cycyle 1 evaluation reports is now serving as a contribution to planning 
revisions for Cycle 2. 

Implications for Developing Countries 

This evaluation project in itself is only of interest if the experiences 
from it can be transferred and applied to the evaluation of other computer- 
related initiatives. A particular value would be the transfer of the 
approach to computer-related projects in less developed countries. 
Constraints of resources as well as exploratory rather than objectives- 
driven activities will characterize many of these initiatives; yet the 
contribution of an evaluation is particularly important when severely 
limited resources cannot afford to be wasted. 
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Based on experience with using the model in The Netherlands as well as 
in other lorge-focu? evaluators, it appears reasonable to advocate the use 
of the model in other national situations involving computer activity in 
education. Th-^ problems that are typically associated with such evaluations 
can be handled within the odel. Thr perception of evaluation as being "too 
early" relative to the maturation level of such activity is not pertinent 
with the moH^i. Even activity that is only in its initial stages can be 
considered relative to the logical associations between its assumptions, 
goals, and currently planneU strategies. Contrary to being seen as a 
judgmental, summative activity, the model encouracjes a conceptuatization of 
evaluation as a constructive, interative process valuable to activity 
inte pretation and modification even as the activity is developing. The 
timely infusion this sort of interp)retive comment is especially helpful 
to exploratory technology-related projects where both environment and 
personnel characteristics can change rapidly during the implementation of 
an activity. The contribution of this approach to evaluation to the process 
of goal formation is also strong. The gradual maturation of computer- 
related activity from a "belief driven" to a "goal driven" process will 
only occur with periodic assessments of what is being learned about the 
beliefs as well as what is really meant by the beliefs. The model 
enphasizes these reconsiderations of assumptions and expectations. 

A final comment must be made about the aspect of having an external 
evdluator stimulate this sort of appraisal. The particular contributions of 
an evaluator from outside the system are that he or she can often better 
identify assumptions and distinguish between assumption-related 
consequences and iit^lementation-related consequences than can individuals 
within the system who are often involved in both the assumptions and their 
execution. Also, with respect to computer-related activity in education. 
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the rapid escalation of experiences in other countries can be helpfully 
applied to the interpretation of local experiences; a broadly based 
external evalator can bring this contribution. There are, of course, 
difficulties in working with an external evaluator, chief among them being 
language and cultural differences. The external evaluator must also be able 
to rapidly appraise the political realities of the situation as well as 
identify the significant decision making chain. For these reasons it is 
helpful if the external evaluator can be teamed with an internal person. 
The internal person must be knowledgeable and bilingual but should not be 
directly part of the computer-related system him or herself. Given this 
sort of support/ the approach diagrammed in Figure 1 can be particularly 
receommended for countries with more limited resources, in that this gives 
them heightened incentive to accumulate the best insights possible about 
the realistic possibilities of utilization of computer-related technology 
in their educational systems. 

References 

Baker/ E.L. (1988/ April) . Sensitive technology assessment of ACOT . Paper 
presented at the Annual Meeting of the American Educational Research 
Association/ New Orleans. 

ColliS/ B./ & Berger, L. (1987). Evaluation Design: POCO Project Cycle 1 . 
Enschede/ The Netherlands: ECC (Educational Computing Consortium). 



\4 




Collis^ B., & Moonen, J. (1988, April). Designing an external evaluation of 
a large scale software development project . Paper presented at the 
annual meeting of the International Association for Computers in 
Education, New Orleans. 

Moonen, J. (1987). POCO Plan of Action . Enschede, The Netherlands: ECC 
(Educational Computing Consortium) . 

Stake, R.E. (1973). The countenance of educational evaluation. In B.R. 
Worthen & J.R. Sanders (Eds.), Educational evaluation; Theorv and 
practice (pp. 106-125). Belmont, CA: Wadsworth Publishing Company. 

Suchman, E. (1976) . Evaluation research . New York: Russell Sage Foundation 
Press. 




15 




